Rank aggregation with ties: Experiments and Analysis

The problem of aggregating multiple rankings into one consensus ranking is an active research topic especially in the database community. Various studies have implemented methods for rank aggregation and may have come up with contradicting conclusions upon which algorithms work best. Comparing such results is cumbersome, as the original studies mixed different approaches and used very different evaluation datasets and metrics. Additionally, in real applications, the rankings to be aggregated may not be permutations where elements are strictly ordered, but they may have ties where some elements are placed at the same position. However, most of the studies have not considered ties. This paper introduces the first large scale study of algorithms for rank aggregation with ties. More precisely, (i) we review rank aggregation algorithms and determine whether or not they can handle ties; (ii) we propose the first implementation to compute the exact solution of the Rank Aggregation with ties problem; (iii) we evaluate algorithms for rank aggregation with ties on a very large panel of both real and carefully generated synthetic datasets; (iv) we provide guidance on the algorithms to be favored depending on dataset features. ∗LRI (Laboratoire de Recherche en Informatique), CNRS UMR 8623 Univ. Paris-Sud France †State Key Laboratory of Virology, College of Life Sciences, Wuhan Univ., Wuhan, China. ‡Univ. Bordeaux, LaBRI, CNRS UMR 5800, F-33400 Talence, France §Inria, Montpellier France ¶I2BC (Institute for Integrative Biology of the Cell), CEA, CNRS, Univ. Paris-Sud France ]DIRO (Département d’Informatique et de Recherche Opérationnelle) Univ. Montréal Québec Canada ?Corresponding authors This work is licensed under the Creative Commons AttributionNonCommercial-NoDerivs 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/. Obtain permission prior to any use beyond those covered by the license. Contact copyright holder by emailing [email protected]. Articles from this volume were invited to present their results at the 41st International Conference on Very Large Data Bases, August 31st September 4th 2015, Kohala Coast, Hawaii. Proceedings of the VLDB Endowment, Vol. 8, No. 11 Copyright 2015 VLDB Endowment 2150-8097/15/07.